Solution

To understand how to update weight matrix U, we will need to unfold the model in time. We will unfold the model over two time steps, as we need to look only time t and time t+1. The following three pictures will help you understand the three paths we need to consider. Notice that we have two hidden layers that serve as memory elements, so this case will be different than the one we saw in the video, but the idea is the same. We will use BPTT while applying the chain rule.

_The first path to consider_ — *The first path to consider*

The following is the equation we derive using the first path:

_The second path to consider_ — *The second path to consider*

The following is the equation we derive using the second path:

_The third path to consider_ — *The third path to consider*

The following is the equation we derive using the third path:

Finally, after considering all three paths, we can derive the correct equation for the purposes of updating weight matrix U, using BPTT:

_The final answer for BPTT Quiz 3_ — *The final answer for BPTT Quiz 3*